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The work which I will be describing is one of the research activities 
being conducted as part of the Projact on Secondary Analysis (PSA) at 
Nor th^jres tarn University* The PSA is being funded by a grant from the 
National Institute of. Education and is "designed to (a) test and develop 
new methods of evaluating educational programs ? and (b) reanalyze eKisting 
evaluation data to assure that estimates of prDgram effects are as unbiased 
and unequivocal as possible*' (Boruch, Wortraans and DeGracie, 1975)* To 
date, reanalyses have been performed on such prograras as Headstart (Magldsons 
19771 Rindskopf and Wolins, 1977) , Title I (Rindskopf and Wolins, 1977), 
ESAP (Alsip, 1977), Middlastart (Boruchp Hagidson, Davis, 1975) and the . 
Alum Rock Voucher EKperiment (Wortman, Reichardtp and St, Pierre, 1977)* 

With few eKceptions, not much systamatic effort has gone into examining 
the topic of secondary analysis* Although there are some good eKamples of 
secondary analysis efforts in social sclanca research such as the Mosteller 
and Mqynihan (1972) volume examining the Coleman data, and Cook's (1975) 
book on the reanalysls of Sesame Street, the unique features of each sec-- 
ondary evaluation make them difficult to classify. 

An eKcellent general exposition of the topic of secondac^y analysis and 
secondary evaluation is presented by Thomas Cook (1974). Cook -a paper 
presents a 2k2x2 classification of secondary analysis models* The eight 
calls in Cook's table are generated by considering whether an analysis 
involves (1) a single data set or multiple data sets, (2) data reanalysls 
or no data^ reanalysls^ (3) ^af ter-the-^f act secondary analysis or simultaneous 
secondary analysis, Cook provides exaraples for each of his eight cells 
but also makes explicit that not all secondary analyses will fit into only 
one cell. - 
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The approach by Rezmovlc and Regmovic (1976) examines one cell In 
Cook-s (1974) matrlK, thm case Involving reanaljrsis of a single data set 
after the fact. This is the situation which hai been cyplcally encoyntered 
in the PSA* ReF.movic and Razmovic also present a 2k2k2 approach to secondary 
analysis considering whether the secondary analyst (1) used new statistical 
methods or repeated the same methods as the original investigator ^ (2) was 
concerned with the same variables as the primary analyst or eKamined different 
variables and (3) whether the reanalysis addressed the same issue as the 
origit^al evaluation or a set of different hypothesis. As with Cook Vs matriK, . 
the Removic and Reimovlc matriK may place a secondary analysis project into 
several cellst The reanalysis of the Call Project for example, encompassed 
matiy csf the cells in the matrix, 

A recent paper by Hendricks and Wortman (1977), although presenting no 
taKonomy, looks at secondary analysis as a decision-making process in which 
the secondary analyst makes key choices dlffarent from those of the original 
invest igator. The main point of this paper is that decisions about analysis 
strategies should not be seen as better o.r worse when compared to one another, 
but that different approaches can answer similar questions of interest. People 
do secondary analysis for a variety of reasons. Housewives , for example^ 
routinely check their grocery bills after shopping at the supermarket to make 
sure thay have not been cheated. University professori delight in examining 
statistical rep&rt a and finding a nmnerical Inconsistency In a data table 
provided by another colleague* or one of their graduate students. Researchers 
reexamine studies conducted by others hoping to find evidence for a theoretical 
position to which they subscribe. The motives and methods of those conducting 
a reanalysis are varied- I would like to mention some of the more common reasons. 



Ttia first atep in performing' secondary analysis frequently InVDlves 
redDing the analyses of the original Investigator* This usually means 
using raw unaggregated data to feproduce meams, variances, correlations, 
and cross tabulations* This is done to insure that the data sent from 
primary investigator to the secondary analyst are the same. It is not 
unusual to hear of data tapes prepared at one Gomputfir installation pro- 
ducing slightly different data wheii output at a second center* Another 
reason for reproducing these basic analyses is to make sure that these 
measures were correctly computed* The quality of results reported in 
Journal articles and published reports la not well known* In one study 
attempting to measure data quality, Woliris (1962) discovered conceptual 
and computational errors in three of seven studies which he reanalyzed. 
Discrepancies between the original results and the reanalysis may be due 
to sources such as keypunching errors/ varying the nuaaber of subjects 
used to compute the statistics 5 or mislabeling certain variables. 

Kesearch is sometimes conducted under time pressure and with limited 
resources in computersi programBs personnel i and finances* Given these 
realistic constraints ^ an evaluation may not be totally adequate, fhe 
secondary analyst J In such an instance* may want to perform different 
analyses which answer the research questions In more depth* One advantage 
of being a secondary analyst is not having to perform under the strict 
time constraints that primary researchers are frequently subjected to, * 

Using statistics to evaluate the success or failure of a program 
Homatlmes Involves arbitrary decisions about which particular staCiBtical 
tc^t to use- Different stntlBClcal methods often pi'ovide esHentlally the 
same information, e.g., using analysis of variance instead of regression. 



However, in some cases statistics can be used inappropriately, resulting 
in different rasults than If the more "correct" method had been used. The 
most coimaon instance of this is the use of a statistic when one or more of 
the assumptions underlying the statistic have been violated* The classic 
paper by Campbell and Irlebacher (1970) demonstrates how regression effects 
may have mads the original Headstart evaluation look harmful* A more recent 
demonstration of this type of problegprs^^provided by--Ri and Wolins 

(1977) J showing how inappropriate use of the analysis of covariance may have 
biased the results of an evaluation of the Title 1 education programs* 

In certain inetances> the motivation behind a secondary analysis may 
be to clarify certain issues which may be vague in the original report* 
For eKmple^ when statistical results are not significant at the .05 level 
the secondary t.. %I^yst may wish to use a statistical test which is more 
powerful to Insure that the lack of significance was not due to a test low 
in power* For example, the analysis of covariance increases the power of 
an F test over that produced by the regular anlaysis of variance model. 
Another eKample might involve reanalyzing a project in which correlations 
were reported but direction of cauaality is unclear. Using a structural 
equation model might provide more information about the direction of cause. 
Other instances of this type of anlaysis might involve testing for inter- 
actions not examined by the original investigator. 

There have been several examples of secondary evaluations which have 
attempted to refute the results presrinted by the primary analyst. The 
most notable contemporary example Is the research on the genetic components 
of intelligence * The analyses by Kamln (19'74) have attempted to refute 
conclusions drawn by Jensen (1969). Another eKample is the Elashoff and 



Snow (1971) reanalysis of Rosenthal and Jacobson (1968) data on teacher 
expectancy. 

Many data sets contain information on variables which are not analyzed, 
or even reported By the primary Investigator* For example ^ a data set may 
contain Jnfonmtion on program participants which relates SES to income 
but these variables may not Interest the original investigator . A secondary 
analyst interested in the relation between income and SES would want this 
data set but his analysis would not involve secondary analysis per se. One 
possibility for primary inveetlgators who have collected large amounts of 
variables which remain unanalyEed is to advertise their data sets so others 
may use the information which is present. Creation of data archives would 
particularly facilitate this type of effort/ 

Many new methods for evaluating programs exist* These methods are 
derived by statisticians and mathematicians and rarely have immediate 
relevance for evaluation researchers * Testing out these methods with 
existing data sets serves a valuable purpose in that members of the 
research community are Informed as to how to use these techniques and what 
the advantages of the techniques may be. In many cases new methods are 
used with simulated data and these have the advantage of letting the 
researcher know the true model underlying the data • However, when using 
empirically generated data certain problems may arise which would not arise 
using the simulated data, 

Reanalysis has the advantage of teaching studep.ts and researchers * 
about how to do research. Analytic methods need to be practiced to be 
understood. Availability of data banks of project information is a valuable 
tool in the teaching process, Currentlyj many courses in rcHcarch mcthodB 
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- uee a data baseband" have students' ■ ' asslgnmerits^^^ 
are performed on this data base. 

When performing secondary analyses on multiple data sets one may be 
able to resolve contradictory findings that plague a research area. For 
example J an education program Implemented at two sites may bring success 
at one and failure at the other* A seeondary analysis might determine what 
factors differ between the sites and how these factors may have prevented 
success at the second site* Another example of this is the meta-analysis 
approach taken by Gene Glass (1976) in which results of studies in an area 
are aggregated to give a, more coherent picture of the findings. 

Integrating contradictory findings may serve a second function, i.e,p 
allowing the secondary analyst to make statements about the policy relevance 
of certain research fesults* Seeondary analysis may come too late to 
provide information for deelsion-iaakers at certain key points in the decision 
making pfoceaa. Accumulated findings,^ however ^ will certainly have an impact 
at later stages as policy Is continually shifting with new administrations* 
Up to now, I^ve addressed general issues in performing a secondary analysis, 
I would now like to direct my conments to the secondary analysis of the Call 
data set In particular • 

Often times the aost difficult part of conduatlng a secondary analysis 
is the act of getting the data ^ In the case of the Call data, many of the 
typical problems which are encountered were not present* This is due. In 
part^ to the long relationship between Northwestern University and the 
Call researchers. Many members of the Northwestern coiranunity have served 
and continue to serve as advisors to the project* This bngQlng relation- 
ship has kept a channel of communication open between Call and Evans ton . 



and when the PSA began it was natural tliat the Call data would be a data 
set which would be reanalyzed. 

Another reason for the smooth relationship between Call and Northwestern 
University lies in the different interests of the two groups. Researchers 
in the Call Projects for examplej are interested in questions relating 
to the effectiveness of the program* Does the treatment improve cognitive 
performance and reduce malnutrition? Their interests center around sub-- 
stantive matters relating to the world of the undernourished Colombian 
child. Those of us involved In the secondary analysis are educational 
researchers whose main Interests concern the development of new methods 
to measure program impact. 

One might argue that our reanalysis is not a true "secondary analysis** 
in that we have dealt with new issues not related to the original analyses. 
Such criticism is partially justified becaus© many of our analyses are 
really primary ones and much of our work might be considered concurrent 
evaluation as opposed to after-the-fact evaluation. In one sense, our 
analyses are extensions of the primary analysis because of the richness of 
the data base. No single research team could hope to use all the information 
collected by this project i 

Based on our experiences with the Call Project we would urge primary 
investigators J who collect evaluation data at great cost and efforts to 
share their data with others so that maximum benefits accrue to the research 
CQtmnunity. 

The PSA has been seeking educational data sets amenable to secondary 
analysis. We feel that the Call data provide us with good opportunities 
to perform various analyses. There are several characteristics of the 
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data base which we find useful in our secondary analysis efforts. 

For exMplei the longitudinal nature of the data* l,e*, repeated 
measurements over a five--year period have allowed us to use within 
subjects analyses which have more statistical power* In addition^ the 
repeated measurement have enabled us to study differing growth profiles 
in the experimental groups as a function of treatment* Since the Cali 
study employed a true eKperlBiental design with random assignments of 
subjects to treatment conditions, it has maximlzad bath the primary and 
secondary analysts' abilities for drawing unequivocal conclusions about 
treatment effects* In addition, its Inclusion of a non-equivalent control 
group enabled us to compare the randomized controls with the non-equivalent 
controls. This comparison Is important in helping to establish the di- 
rection and magnitude of bias produced when non-equivalent controls are 
utillEed, the typical case In most social experiments. 

Multiple measures of achievement | SESp and medical status are valuable 
because they facilitate the use of many sophisticated techniques which 
require multiple meaaurements of a single latent variable. These multiple 
measures also allow us to measure the effects of the program In more 
specific terms and to outline the relative success of the treatment in 
various contexts of the child's modes of performance. 

We must applaud the members of the project for the care taken to 
Insure that the quality and documentation of data were adequate for our 
purposes. Although we received over 130 data cards for each of over 300 
cases, there has been no difficulty locating the information of interest* 
The data system Instituted In Call has been very succeisful and much eSffort 
has been devoted to minimizing both transcription and keypunching errors* 
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Our initial efforts of the secondary analysis entailed replicating 
the original means 5 variances, and correlations produced by the original 
Call rasearchere* We were able to do this successfully and were quite 
certain that our results matched theirs* Although we did discover very 
minor errors in keypunching , such as values which exceeded the inaximum 
possible scorei these errors were so infrequent and affected the results 
so slightly^ that we were confident about the results of the primary report. 

We are happy to say that all the methods that we used for measuring 
program effect led to conclusions which were similar to those of Dr, McKay 
and his colleagues* 

Our analyses which included repeated measures ANOVAj univariate and 
multivariate ANOVA, and diicriminant analysis ^ all supported the conclusion 
that the program works. These strong results allowed ue to concentrate on 
questions of methodological interest rather than trying to establish how 
well the program worked. 

Many sophisticated methods of data anlaysis eKist which can be used 
to assess the effects of treatment programs. Frequently, these methods 
are first described by mathematical statisticians and are not inmediately 
available to educational researchers* This is either because the computer 
software to perform these analyses is not available or because these 
techniques have not been reliably demonstrated in educational contexts* 
One of the aims of the PSA has been to demonstrate the applications of . 
new methods of analysis to conventional educational research situations, 
I would like to mention several of these* 

Our analyses looked at the Call data of height and weight as repeated 
measuremehts over a five-^year period using seven bbservations during this time 
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In dealing with the issue of repeated measures analysiSj two approaches 
were contrasted. The univariate analysis of rapeated measures (or mixed 
models as it is also known) looks at whether profiles for the various 
treatment groups are parallel* If so, we conclude that there was a 
treatment affect. In the multivariate approach, we treat the rapeated 
observations as a set of dependent variables and the difference between the 
groups can be tested. The univariate approach makes the assumption that 
the pooled within-group ©Khibits compound symmetry, i.e., that the same 
construct is being measured. at each time point with equal reliability. The 
multivariate approach makes no assmnptions about E* The difference between 
the two approachas Is that when the assumptions of the univariate approach 
hold it yields a more powerful analysis. In the multivariate approach, many 
degrees of freedom must be used at estimating the wlthin-group covariance 
matrix. In our reanalyslSj we performed both types of analysis and examined 
how the univariate statistics could be extracted from the multivariate 
results. 

Before the Rasch model approach was taken, an aim of the original Call 
analysis was to combine several cognitive measurements into a general ability 
factor and to compare groups across the time periods on this factor. This 
approach s while included in an earlier report, was later rejected because 
it did not consider the question of factorial invarlance between groups and 
across occasions. Recently Joreskog (1971) has described a technique for 
testing factorial invarlance. Using Joreskog's technique, we focused on 
the question of factorial Invarlance among treatment groups at the last 
treatment period. Analyses were performed which examined the factorial- 
structure of the WISC-R subtests used in five randomly assigned Call groups. 



Results of this analysis yielded a stable factor pattern with two factors 
(verbal and per formance) Invariant among all groups* Using the Joreskog 
approach J we were also able to compute group factor means. 
V Two problaras wfcich have plagued researchers concern the effect of 
measurCTent error on statistical results and (2) making causal inferences 
in correlational research* There has been a recent interest in the area 
of structural equation models which deal with these two Issues* In par- 
ticulari we were Interested in examining the causal relationship between 
nutritional status and intelligence* A causal model was hypothesized in 
which we could examine the effects of nutritional status on intelligence. 
The next presentation will examine some of these analyses in more detail* 

Our reanalysis of the Call data attempted to use new statistical tech-- 
niques to answer questions of educational signlf Icance , and to provide the 
research coxmnunity with examples of new methods which can be used in evalu- 
ation contexts. We hope our efforts will encourage other educational 
researchers to further these goals- 
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Footnotes 

1* Work on this manuscript was iupported by Contract No* 
from the National Institute of Education. 
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